A Head-to-Head Comparison of Human-Based and Automated Text Analysis for Measuring Populism in 27 Countries∗

In this paper we use holistic grading to measure the level of populism found in political parties’ discourse during electoral campaigns. The technique is applied to electoral manifestos from 144 parties in 27 countries from Western Europe and the Americas. For about half the sample, we also code speeches by the candidates or party leaders, which indicate the validity of measuring populism on electoral manifestos. Our results indicate that populism is stronger in Latin America than in Europe, and that the level of populism among some European parties typically considered as examples of radical right populism might be overstated when compared with Latin American counterparts. We contrast these results with automated content analysis of the manifestos through machine learning. In this sample, the results of computerized text analysis are not satisfactory, but indicate that with more data it could be a viable option. ∗An earlier version of this paper was presented at Explaining Populism: Team Populism January Conference, Provo-UT, January 28-3